Freehand Sketch Recognition for Computer-Assisted Language Learning of Written East Asian Languages

One of the challenges students face in studying an East Asian (EA) language (e.g., Chinese, Japanese, and Korean) as a second language is mastering their selected language’s written component. This is especially true for students with native fluency of English and deficient written fluency of another EA language. In order to alleviate the steep learning curve inherent in the properties of EA languages’ complicated writing scripts, language instructors conventionally introduce various written techniques such as stroke order and direction to allow students to study writing scripts in a systematic fashion. Yet, despite the advantages gained from written technique instruction, the physical presence of the language instructor in conventional instruction is still highly desirable during the learning process; not only does it allow instructors to offer valuable real-time critique and feedback interaction on students’ writings, but it also allows instructors to correct students’ bad writing habits that would impede mastery of the written language if not caught early in the learning process. The current generation of computer-assisted language learning (CALL) applications specific to written EA languages have therefore strived to incorporate writing-capable modalities in order to allow students to emulate their studies outside the classroom setting. Several factors such as constrained writing styles, and weak feedback and assessment capabilities limit these existing applications and their employed techniques from closely mimicking the benefits that language instructors continue to offer. In this thesis, I describe my geometric-based sketch recognition approach to several writing scripts in the EA languages while addressing the issues that plague existing CALL applications and the handwriting recognition techniques that they utilize. The approach takes advantage of A Language to Describe, Display, and Editing in Sketch Recognition (LADDER) framework to provide users with valuable feedback and assessment that not only recognizes the visual correctness of students’ written EA Language writings, but also critiques the technical correctness of their stroke order and direction. Furthermore, my approach provides recognition independent of writing style that allows students to learn with natural writing through size- and amount-independence, thus bridging the gap between beginner applications that only recognize single-square input and expert tools that lack written technique critique

A Sketch Recognition-based Intelligent Tutoring System for Richer Instructor-like Feedback on Chinese Characters

Foreign language courses in the East Asian languages of Chinese, Japanese, and Korean (CJK) have been popular choices for study and enrollment in the United States for various reasons. Students in particular express a range of interests from enhancing their career opportunities in the region’s rising business and technology sectors to greater appreciating those countries’ traditional and popular cultures. For such students learning a CJK language as a foreign language and with only primary fluency in English, mastery of a CJK foreign language is challenging due to vastly distinct linguistic differences. These challenges are especially true for the CJK languages’ written component, whose Chinese characters (i.e., written script logograms used in the CJK languages) are vastly more difficult to achieve fluency in writing compared to the smaller, less varied, and less complex set of alphabet letters used in the English language. To address the steeper learning curve that students from the United States experience with written CJK languages, instructors conventionally introduce varying learning strategies (e.g., rote writing memorization and stroke writing technique) and individualized assessment and feedback of students’ writing performance. With learning situations that lack or reduce exposure to language instructors and their invaluable personalized responses to students’ writing performance (e.g., outside classroom hours, larger-sized classrooms, self-taught study), complementary intelligent tutoring systems (ITS) have grown in sophistication to emulate the successful strategies of their human language instructor counterparts and to also encourage greater consistency in assessment among students. However, despite improved recognition capabilities and interface design of such ITS interfaces for introductory written CJK instruction, the current state of these ITS interfaces have yet to capture more granular feedback provided by human language instructors. More specifically, current ITS interfaces are limited on their reliance of either binary feedback or simplified assessment of language students’ writing performance. Without richer assessment levels for language students to gauge their writing performance, they are more prone to develop incorrect writing habits that are detrimental to their study of written Chinese characters. In this dissertation work, a sketch recognition-based ITS interface is proposed for providing richer assessment and feedback that emulates human language instructors, specifically for novice students’ introductory course study of the CJK languages and their written Chinese characters. To capture finer granularity of feedback from human language instructors and to also address the assessment limitations of existing intelligent tutoring systems for introductory characters, the work aims to explore the following three goals: 1. Discover a list of successful assessment techniques that human language instructors utilize in classroom instruction for character writing and practice. 2. Design an interface displaying visualization and feedback cues that students can intuitively understand and measure their writing performance. 3. Develop a set of automated sketch and handwriting recognition techniques to accurately capture those assessment strategies. From the proposed ITS interface, a stylus-driven solution is provided to novice language students for studying and practicing introductory Chinese characters with deeper assessment levels, so that they may have richer feedback to improve their writing performance. In order to evaluate the performance of the interface, three studies—a classroom study, an interaction study, and an assessment study—was conducted to measure the interface’s effectiveness in learning in the classroom, intuitiveness in interface usage, and comparisons with instructor assumptions, respectively. The results of the evaluations demonstrate that the interface was effective in the investigated categories, and that it was successful in its purpose as a writing interface for Chinese character writing and practice